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EXECUTIVE  SUMMARY 


Until  recently,  little  attention  has  been  paid  to  the  procedures  used  to  develop  and  revise  aircraft 
maintenance  technical  data.  Studies  of  maintenance  errors  have  tended  to  focus  on  the  actions  of 
the  mechanic,  job  culture,  and  work  procedures.  More  recently,  attempts  have  been  made  to 
document  the  source  of  meiintenance  errors  and  improve  maintenance  procedures.  One  of  the 
identified  contributing  causes  of  errors  is  the  documentation  used  to  guide  maintenance  tasks. 
As  a  result,  efforts  have  been  made  to  establish  guidelines  for  the  design  of  maintenance  job  aids. 
A  question  that  remains  is  how  the  procedures  used  by  manufacturers  to  develop  maintenance 
data  may  contribute  to  user  error. 

In  this  document,  the  results  of  Phase  1  of  a  three-phase  research  effort  to  (1)  examine  the 
procedures  used  by  industry  to  develop  maintenance  manuals,  (2)  document  the  problems 
encountered  by  users  of  these  documents,  and  (3)  identify  ways  in  which  human  factors 
principles  can  be  used  to  improve  the  development  of  these  documents  are  reported.  Phase  1  is  a 
survey  and  report  of  the  procedures  used  within  the  aviation  industry  to  develop  maintenance 
technical  data.  A  cross-section  of  manufacturers  was  surveyed  regarding  company  policy, 
communication,  data  tracking,  user  feedback,  and  error  reduction  efforts. 


v/vi 


1.  INTRODUCTION. 


1.1  BACKGROUND. 

A  little  known  fact  is  that  aircraft  manufacturers  rank  among  the  largest  publishers  in  the  world. 
With  each  aircraft  they  produce,  they  provide  the  technical  documentation  needed  to  maintain  the 
aircraft  in  working  order.  Aircraft  manufacturers  provide  a  wide  range  of  documentation 
including  the  maintenance  manual  itself,  an  illustrated  parts  catalog,  wiring  diagrams,  structural 
repair  manuals,  and  a  host  of  other  related  documents.  The  development  and  revision  of  aircraft 
maintenance  technical  manuals  is  no  small  task.  The  amount  of  information  is  staggering,  with 
manuals  being  measured  more  appropriately  by  the  feet  of  shelf  space  they  occupy  rather  than  the 
number  of  pages  they  contain.  For  aircraft  that  have  any  degree  of  customization,  the 
manufacturer  may  need  to  develop  different,  operator-specific  manuals  to  include  only  the 
information  relevant  to  the  aircraft  they  maintain.  The  Boeing  Company  estimates  that  if  all 
manual  pages  published  in  1998  were  stacked  one  on  top  of  another,  the  resulting  tower  would 
reach  approximately  103,000  feet  in  the  air  [1]. 

The  size  and  complexity  of  a  modem  technical  manual  requires  the  integration  of  information 
from  multiple  sources,  making  it  increasingly  difficult  to  verify  the  accuracy  and  clarity  of  the 
information  maintenance  manuals  contain.  Each  company  has  developed  their  own  techniques 
and  procedures  to  deal  with  these  issues,  but  there  is  little  documentation  in  the  public  domain  or 
industry  dialogue  about  the  different  procedures  employed.  While  manufacturers  may  be  hesitant 
to  openly  share  this  information,  the  potential  benefits  include  the  reduced  cost  of  developing 
manuals  and  improved  manual  quality.  Considering  that  the  goal  of  a  technical  manual  is  to 
facilitate  the  safe  and  efficient  maintenance  of  an  aircraft,  it  is  important  to  continually  evaluate 
the  degree  to  which  the  applied  procedures  meet  these  objectives.  The  goal  of  this  effort  is  to 
document  the  unique  ways  companies  have  addressed  the  problems  of  manual  development  and 
to  encourage  a  dialogue  between  the  manufacturers,  operators,  and  regulatory  bodies  within  the 
industry. 

The  task  of  developing  manuals  has  become  a  more  critical  part  of  aircraft  maintenance  in  recent 
years  as  the  demands  being  placed  on  those  manuals  increase.  The  continued  growth  in 
worldwide  air  traffic  has  resulted  in  increased  production  of  new  aircraft  and  older  aircraft 
remaining  in  service  much  longer  than  originally  expected.  While  the  number  of  aircraft  has 
increased,  the  number  of  maintenance  technicians  has  not;  consequently  there  is  greater  pressure 
on  aircraft  maintenance  in  general.  A  1993  Blue  Ribbon  Study,  “Pilots  and  Aircraft  Maintenance 
Technicians  for  the  21st  Century;  An  Assessment  of  Availability  and  Quality,”  found  that 
although  there  were  enough  aircraft  mechanics  at  that  time,  the  projected  demand  was  increasing 
at  a  faster  rate  than  the  number  of  new  mechanics.  Amid  the  favorable  economic  conditions 
experienced  globally  in  the  years  since  that  study  was  released,  the  situation  has  worsened  more 
quickly  than  predicted  and  air  traffic  continues  to  increase.  The  growth  of  air  carrier  operations 
combined  with  a  retiring  maintenance  workforce  and  a  worker-friendly  job  market  has  resulted  in 
a  critical  shortage  of  qualified  aircraft  mechanics.  The  Labor  Department  reports  that 
approximately  140,000  mechanics  work  in  the  aviation  industry  today  and  40,000  additional 
mechanics  will  be  needed  by  2008.  In  order  to  fill  the  positions  necessary  to  support  larger 
fleets,  maintenance  facilities  are  being  forced  to  lower  hiring  requirements  for  maintenance 


1 


personnel.  As  an  example,  American  Airlines  has  recently  been  forced  to  lower  experience 
requirements  from  4  to  2  years.  At  the  same  time,  there  appears  to  be  a  general  lack  of  skilled 
applicants.  In  1997,  responding  to  concerns  about  the  skill  level  of  newly  hired  mechanics. 
United  Airlines  incorporated  a  basic  skills  test  into  their  interview  process  for  maintenance 
technicians.  This  skills  test  involves  tasks  required  of  any  aircraft  mechanic,  including  rivet 
installation  and  safety  wiring  of  bolts.  In  an  interview  with  USA  Today,  the  United  Airlines 
Director  of  Maintenance  reported  that  from  the  end  of  1999  to  mid-2000,  1,600  potential 
employees  were  interviewed,  and  only  45%  of  those  could  pass  the  basic  skills  test  [2].  United  is 
one  of  only  five  airlines  that  include  a  skills  test  in  their  hiring  process.  The  experience  of 
United  Airlines  may  indicate  a  need  for  long-term  changes  in  the  training  of  new  mechanics,  but 
the  short-term  reality  is  that  the  qualification  of  the  maintenance  workforce  appears  to  be 
declining. 

Lesser-trained  and  lesser-educated  maintenance  personnel  will  rely  much  more  heavily  on  the 
content  of  maintenance  manuals.  They  will  also  lack  the  knowledge  that  comes  with  experience 
to  determine  when  a  manual  may  be  in  error.  In  some  cases,  in  order  to  meet  demand,  work  has 
been  offloaded  to  foreign  maintenance  facilities  that  bring  with  them  the  added  concern  of  the 
translatability  of  documents.  Growing  liability  and  regulatory  pressures  have  also  increased  the 
scrutiny  of  technical  manuals  and,  in  some  cases,  have  forced  the  inclusion  of  corporate  lawyers 
into  the  manual  development  process.  Because  of  these  pressures,  decisions  regarding  reading 
level,  writing  style,  clarity,  and  the  degree  of  detail  to  be  used  when  writing  maintenance 
manuals  are  even  more  important  now  than  in  the  past. 

The  way  a  particular  manufacturer  may  handle  these  content  decisions  is  often  a  matter  of 
established  company  history.  Manufacturers  develop  a  writing  style  that  is  propagated  from 
veteran  writers  in  the  company  to  the  newly  hired  writers,  and  users  come  to  expect  a  certain 
style  from  a  particular  manufacturer.  Style  consistency  may  have  the  benefit  of  a  perceived 
familiarity  across  manufacturers  models,  but  it  may  not  be  appropriate  for  the  changing  face  of 
the  maintenance  workforce. 

In  1994,  a  review  of  major  aircraft  accidents  found  that  approximately  12%  could  be  attributed  to 
maintenance  [3].  According  to  the  Boeing  Company’s  summary  of  commercial  airplane 
accidents  worldwide  for  the  years  1959-1999,  the  figure  is  a  more  conservative  5.9%  [4]. 
Regardless  of  which  figure  one  chooses  to  use,  there  has  been  an  apparent  increase  in 
maintenance-related  aircraft  accidents  in  recent  years.  While  aircraft  accidents  capture  headlines, 
the  truth  is  that  few  maintenance  errors  ever  result  in  accidents.  Analysis  of  maintenance  errors 
indicates  that  the  majority  of  incidents  involve  omitted  or  incorrect  execution  of  tasks,  incorrect 
installations,  and  the  use  of  incorrect  parts  (Boeing  research  cited  in  reference  5).  Such  events 
can  potentially  lead  to  accidents  if  left  uncorrected,  but  these  are  rare  events,  and  to  focus  on 
accidents  alone  is  to  underestimate  the  actual  number  of  errors  and  their  impact  on  safety  and 
operating  costs. 


A  Human  Factors  approach  to  reducing  maintenance  error  requires  that  causal  attribution  be 
extended  beyond  just  the  offending  mechanic.  If  the  search  for  error  sources  is  extended  to 
environmental  influences,  causal  attribution  may  include  latent  errors  introduced  by  management 
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policy,  organizational  communication,  or  corporate  culture  of  the  maintenance  facility.  Another 
potential  factor  in  maintenance  error  that  is  easily  overlooked  is  the  technical  information  used  to 
guide  maintenance  operations.  The  potential  contribution  of  technical  documentation  to 
maintenance  error  is  not  a  new  concern  [6].  However,  previous  attempts  at  improving 
maintenance  documents,  such  as  Drury  and  Sarac’s  document  design  aid  [7],  have  yet  to  be 
adopted  by  aircraft  manufacturers  on  a  large  scale. 

Maintenance  manuals  can  contribute  to  maintenance  error  if  they  contain  misleading  information, 
insufficient  information,  or  unclear  procedures.  Not  only  must  the  information  be  technically 
sound,  it  must  also  be  presented  in  an  effective  manner.  A  term  common  to  the  computer 
industry  most  applicable  in  this  case  is  “usability.”  The  Institute  of  Electrical  and  Electronics 
Engineers  (TREE)  defines  usability  as  “the  ease  with  which  a  user  can  learn  to  operate,  prepare 
inputs  for,  and  interpret  outputs  of  a  system  or  component”  [8].  Applied  to  aircraft  manuals, 
usability  includes  the  user  experience  of  those  manuals;  how  easy  they  are  to  use,  how  well  they 
match  the  mechanic’s  representation  of  a  task,  and  how  useful  the  information  is  they  contain. 
The  user  experience  of  the  manuals  will  also  impact  the  ways  and  degree  to  which  mechanics 
will  use  them.  In  a  study  conducted  for  the  Australian  Transportation  Safety  Bureau,  67%  report 
having  been  misled  by  maintenance  documentation,  47%  report  having  opted  to  perform  a 
maintenance  procedure  in  a  way  they  felt  was  superior  to  that  described  by  the  manual,  and  73% 
of  mechanics  surveyed  reported  failing  to  refer  to  maintenance  documents  either  occasionally  or 
often  [9]. 

These  findings  raise  concerns  about  the  perceived  usability  of  manuals  and  the  quality  of 
information  exchange  between  mechanics  and  the  writers  of  maintenance  materials.  The  fact 
that  mechanics  report  instances  of  failing  to  refer  to  maintenance  manuals  may  be  indicative  of 
familiarity  with  the  task  or  prior  experience  with  manuals  that  were  hard  to  use  or  not  very 
helpful.  If  user  comments  are  relied  on  as  the  final  check  of  manual  quality,  the  usage  pattern 
becomes  a  critical  issue.  A  lack  of  user  complaints  may  be  indicative  of  a  well-crafted  manual  or 
one  that  users  have  simply  stopped  using  and  see  no  point  in  bringing  these  issues  to  the  attention 
of  the  developer. 

The  task  of  developing  and  revising  maintenance  manuals  requires  the  coordination  of  multiple 
information  sources  across  a  number  of  departments  within  the  management  structure  of  the 
manufacturer.  Engineering,  technical  support,  customer  service,  and  technical  writing  must 
integrate  the  most  recent  information  from  their  respective  sources  to  provide  the  technical  base 
necessary  to  produce  a  technically  sound  document.  Technical  writers  have  the  ultimate 
responsibility  of  verifying  that  they  have  the  most  recent  and  accurate  information  available  on 
which  to  base  technical  manuals. 

The  accuracy  of  the  information  contained  in  maintenance  manuals  is  of  paramount  importance, 
and  manufacturers  have  implemented  multiple  safeguards  to  protect  against  the  inclusion  of 
erroneous  content.  Document  checklists,  peer  review,  and  software  formatting  have  all  been 
implemented  to  reduce  the  number  of  errors  present  in  fielded  maintenance  manuals.  Once 
released,  each  manufacturer  has  procedures  for  addressing  problems  that  users  encounter  while 
using  the  manuals.  The  accuracy  of  maintenance  manuals  gets  considerable  attention  from 
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manufacturers  and  operators,  and  the  continued  application  of  computer  technology  to  the 
technical  writing  task  promises  greater  ability  to  verify  the  accuracy  of  technical  information. 

1.2  REGULATORY  REQUIREMENTS. 

The  current  procedure  for  developing  maintenance  manuals  is  an  integral  part  of  satisfying  the 
Federal  Aviation  Administration  (FAA)  requirement  for  operators  to  develop  and  support  an 
ongoing  maintenance  program.  More  specifically,  manuals  are  designed  to  help  operators  meet 
the  requirements  outlined  by  the  following  Federal  Aviation  Regulations  (FAR). 

•  FAR  Part  121.363  establishes  the  operators  responsibility  to  maintain  the  airworthiness  of 
its  aircraft  fleet. 

•  FAR  Part  121.367  requires  each  certificate  holder  to  have  programs  for  aircraft 
inspection,  preventive  maintenance,  and  oversight  of  alterations  that  ensures  these  tasks 
are  performed  in  accordance  with  the  certificate  holders  manual.  The  certificate  holder  is 
further  required  to  ensure  that  these  tasks  are  performed  by  competent  personnel  with 
adequate  facilities  and  equipment.  Finally,  the  certificate  holder  is  ultimately  responsible 
to  ensure  that  each  aircraft  released  to  service  is  airworthy  and  properly  maintained. 

•  FAR  Part  121.373  requires  each  certificate  holder  to  establish  and  maintain  a  program  to 
monitor  the  performance  and  effectiveness  of  its  inspection  and  maintenance  program. 
The  certificate  holder  is  responsible  for  correcting  any  identified  deficiencies  in  those 
programs. 

•  FAR  Part  121.379  establishes  the  authority  of  an  air  carrier  to  perform  aircraft 
maintenance  in  accordance  with  an  FAA-accepted  maintenance  manual.  The 
maintenance  can  be  performed  by  the  carrier  itself  or  another  approved  facility.  If 
maintenance  requires  a  major  repair  or  major  alteration,  the  work  must  have  been 
accomplished  in  accordance  with  FAA-approved  technical  data. 

Manufacturers  are  likewise  required  to  provide  the  technical  instruction  necessary  to  support 
continued  airworthiness  of  their  aircraft.  This  obligation  is  outlined  in  FAR  Part  25. 

•  FAR  Part  25.1529  requires  instructions  for  continued  airworthiness  as  part  of  type 
certification.  The  instructions  may  be  incomplete  at  type  certification  if  a  program  exists 
to  ensure  their  completion  prior  to  delivery  of  the  first  airplane  or  issuance  of  a  standard 
certificate  of  airworthiness,  whichever  occurs  later. 

•  FAR  Part  25:  Appendix  H  elaborates  on  specifications  for  the  technical  information 
required  by  FAR  Part  25.1529.  The  instructions  must  be  written  in  English,  providing  for 
practical  arrangement,  and  in  a  form  appropriate  for  distribution.  Instructions  must 
include  information  about  all  equipment  installed  on  the  aircraft,  including  equipment 
made  by  third  party  manufacturers.  Manual  content  requirements  are  outlined  for  system 
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descriptions,  maintenance  and  inspection  procedures,  required  schedules,  and  information 
about  system  tests  and  service  points. 

While  the  FAA  has  established  a  need  for  airframe  maintenance  manuals  in  a  required 
maintenance  program,  exact  regulatory  requirements  for  those  manuals  are  not  outlined.  The 
manuals  must  be  accepted  as  part  of  the  maintenance  program,  but  the  manual  data  itself  is  not 
approved.  Only  portions  of  the  maintenance  technical  information  require  direct  regulatory 
approval  (e.g.,  Stmctural  Repair  Manual)  while  the  majority  of  maintenance  information  does 
not.  Considering  the  massive  amount  of  maintenance  information  and  the  potential  degree  of 
variation  in  manuals  from  one  operator  to  another,  direct  regulatory  approval  of  all  procedures  is 
not  desirable.  From  the  regulatory  perspective,  the  intent  of  a  maintenance  procedure  is  more 
important  than  the  particulars  of  its  execution.  However,  the  lack  of  detailed  standards 
underscores  the  responsibility  placed  on  the  manufacturer  and,  to  a  lesser  extent,  the  operator  to 
develop  and  maintain  the  quality  of  information  contained  in  manuals. 

2.  DISCUSSION. 

2. 1  MAINTENANCE  TECHNICAL  MANUAL  CONSADERATIONS . 

2.1.1  Development. 

New  manuals  begin  development  during  the  early  stages  of  the  aircraft  design  process.  During 
the  design  and  development  stages  of  a  new  aircraft  program  for  FAR  Part  121  operators,  the 
manufacturer  produces  a  Maintenance  Planning  Document  (MPD)  outlining  the  inspection  and 
maintenance  schedules  that  will  be  required  by  the  airframe  and  system  components.  The 
development  of  the  MPD  is  guided,  in  part,  by  the  Maintenance  Review  Board  (MRB)  document 
developed  by  the  FAA  and  industry  to  aid  in  establishing  initial  inspection  and  maintenance 
schedules.  The  process  is  further  aided  by  logic  developed  through  the  Maintenance  Steering 
Group  (MSG),  an  industry  collaboration  of  manufacturers,  airlines,  and  regulatory  authorities. 
The  MSG  allows  for  the  adaptation  of  the  MRB  process  to  meet  the  unique  requirements  of  a 
particular  aircraft  while  conforming  to  the  original  intent  of  FAR  inspection  and  repair 
regulation.  The  MSG  logic  has  gone  through  several  iterations  and  is  now  referred  to  as  MSG-3. 
The  entire  MRB  process  is  required  for  the  aircraft  to  receive  a  Type  Certificate  and  serves  as  an 
initial  check  of  the  integrity  and  accuracy  of  maintenance  information. 

With  the  required  inspection  and  maintenance  schedules  established,  the  manufacturer  must 
begin  the  task  of  outlining  those  procedures.  In  addition  to  the  required  scheduled  maintenance, 
procedures  must  also  be  developed  for  the  removal,  installation,  and  repair  or  replacement  of  all 
components  installed  in  the  aircraft.  It  is  in  the  outlining  and  description  of  these  procedures  that 
technical  writers  have  the  greatest  freedom  and  it  is  in  this  area  that  manuals  vary  the  most. 

A  guiding  force  in  the  style  and  appearance  of  technical  manuals  is  the  Air  Transport  Association 
(ATA),  through  the  document  guidelines  they  develop.  The  ATA  specification  100,  and  the  new 
ispec  2200  that  replaces  it,  provides  direction  for  the  format  and  layout  of  technical  manuals. 
Although  not  regulatory  nor  mandatory  in  nature,  manufacturers,  operators,  and  regulatory 
agencies  have  come  together  through  the  ATA  to  standardize  the  format  of  technical  manuals. 
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This  standardization  extends  from  chapter  organization  to  headings  and  fonts.  The  goal  is  to 
make  navigation  and  use  of  manuals  as  consistent  as  possible  across  the  industry.  Once  again, 
although  there  is  considerable  structure  imposed  by  the  ATA  specifications,  decisions  about  the 
content  are  left  to  the  discretion  of  the  technical  publications  group.  For  example,  manuals  can 
differ  greatly  in  the  use  of  constrained  language  (i.e.,  restricted  vocabulary  set  and/or  sentence 
structure).  Technical  writers  have  used  constrained  languages  in  an  attempt  to  improve  usability 
by  addressing  the  effects  of  differences  in  reading  level  and  language  fluency  among 
maintenance  personnel.  Some  manufacturers  use  the  United  States  Air  Force  dictionary  that 
limits  the  vocabulary  to  words  that  should  be  understood  by  a  person  with  an  eighth  grade 
reading  level.  Other  manufacturers  use  more  restrictive  standards  such  as  simplified  English,  to 
control  both  vocabulary  and  sentence  structure.  The  simplified  English  standard  has  writing 
rules  and  a  list  of  acceptable  words,  noun  phrases,  and  verbs.  The  use  of  simplified  English  is 
meant  to  eliminate  slang  and  create  concise  sentences.  Because  maintenance  documentation  is 
authored  in  English  and  normally  not  translated  into  the  native  language  of  an  international 
operator,  simplified  English  may  make  it  easier  for  non-native  English  speakers  to  use  technical 
manuals.  Given  all  of  these  factors,  two  manuals,  both  conforming  to  ATA  specifications,  may 
differ  greatly  in  clarity,  reading  level,  degree  of  detail,  and  writing  style. 

2.1.2  Time. 

The  underlying  time  and  budget  pressures  that  drive  the  manufacturers  of  aircraft  need  to  be  kept 
in  mind  when  examining  the  scope  of  the  task  facing  the  writers  of  technical  maintenance 
information.  Safety  is  of  ultimate  concern  to  anyone  in  the  aviation  industry,  but  there  are  often 
trade-offs  between  ideal  safety  practices  and  the  real-world  compromises  adopted.  One  of  the 
single  greatest  factors  in  determining  the  amount  of  money  made  on  an  aircraft  is  time.  An 
airplane  sitting  on  the  production  line  is  not  making  money.  The  longer  it  takes  to  deliver  an 
aircraft,  the  longer  the  delay  until  it  begins  earning  revenue.  Operators  pressure  the  manufacturer 
to  meet  an  established  contractual  delivery  schedule  because,  in  most  cases,  the  new  airplane  has 
already  been  scheduled  for  revenue  service.  This  economic  reality  is  crucial  to  the  full 
appreciation  of  the  environment  surrounding  the  development  of  technical  documents. 

The  time  in  production  determines  the  number  of  aircraft  that  can  be  produced  in  a  year  and  may 
be  the  deciding  factor  in  a  customer’s  choice  of  a  manufacturer.  In  theory,  delivery  could  be 
delayed  to  ensure  that  maintenance  manuals  are  completed.  In  reality,  the  technical  publications 
department  within  manufacturers  is  rarely  given  authority  commensurate  with  engineering, 
production,  or  flight  test  departments.  Delivery  will  not  occur  until  all  final  assembly  on  all 
systems  have  been  installed  and  tested,  but  it  is  unlikely  to  ever  be  delayed  for  the  technical 
documentation  that  refers  to  those  systems.  When  budgetary  concerns  arise,  the  technical 
publications  department  is  often  faced  with  funding  cuts  and  time  limitations.  In  most  cases,  the 
technical  publications  department  is  aware  of  the  problems  they  face  but  lack  the  perceived 
importance  within  the  company  hierarchy  to  command  the  time  and  resources  to  implement 
changes. 


6 


2.1.3  Variations  Between  Manuals. 


During  the  production  life  of  an  aircraft  model,  engineering  and  equipment  improvements 
continue  to  be  made.  Vendors  supplying  parts  for  that  aircraft  are  also  constantly  working  to 
improve  their  products.  These  improvements  are  incorporated  into  production  aircraft  as  they 
become  available.  Because  of  the  evolution  of  the  aircraft  design,  different  serializations  of  the 
same  aircraft  model  may  include  different  parts  and,  therefore,  require  different  maintenance 
procedures.  For  any  given  aircraft  model,  there  is  also  a  variety  of  optional  equipment  that  can 
be  installed.  For  smaller,  regional  aircraft,  configuration  differences  may  be  limited  to  interiors 
and  avionics  options.  For  large  transport  aircraft,  configuration  options  can  be  extended  to 
accommodate  any  desire  of  the  customer  including  engines,  environmental  systems,  and  in-flight 
entertainment  equipment.  The  result  is  that  any  two  examples  of  a  particular  aircraft  model  may 
vary  significantly.  The  mechanic  responsible  for  maintaining  those  aircraft  must  be  provided 
with  the  necessary  information  to  make  the  distinctions  between  the  aircraft  and  adjust 
accordingly  to  the  task  of  repairing  each  of  them.  To  make  this  as  easy  as  possible,  the  manual 
must  contain  all  of  the  information  relevant  to  the  aircraft  the  mechanic  is  working  on,  while  at 
the  same  time  not  require  sorting  through  a  lot  of  nonapplicable  information. 

For  smaller  aircraft,  or  those  with  fewer  customization  options,  it  may  be  acceptable  to  include 
all  information  into  every  manual  and  give  the  user  the  necessary  data  to  determine  what  applies 
to  their  particular  aircraft.  For  more  complex  aircraft,  or  those  with  a  large  number  of 
customization  options,  it  becomes  necessary  for  the  manufacturer  to  tailor  a  customer’s  manual 
to  the  particular  configuration  of  their  aircraft.  When  a  large  transport  aircraft  has  a  maintenance 
manual  numbering  in  the  tens  of  thousands  of  pages,  unnecessary  information  is  not  tolerated  by 
the  operator.  For  these  aircraft,  manufacturers  are  forced  to  provide  manuals  customized  to  each 
operator.  The  way  this  is  handled  depends  on  the  manufacturer. 

The  task  of  the  manual  technical  writer  is  to  obtain  all  of  the  information  necessary  to 
incorporate  both  the  constantly  changing  engineering  information  and  any  necessary 
customization  information  into  the  finished  manual.  Several  sources  within  the  organizational 
structure  of  the  manufacturer  need  to  coordinate  their  respective  data  in  order  for  this  information 
to  make  it  into  the  manual.  Significant  design  changes  affecting  the  form,  fit,  or  function  of  the 
aircraft  must  be  communicated  to  all  affected  internal  organizations.  When  notified  of  a 
significant  design  change,  technical  writers  must  modify  the  applicable  maintenance  data  and 
incorporate  that  information  into  a  manual  revision.  The  burden  is  on  the  technical  writer  to 
ensure  that  the  data  presented  in  the  manual  reflects  the  actual  configuration  of  the  airplane. 

Individual  writers  are  assigned  to  writing  tasks  based  on  their  expertise  with  aircraft  subsystems. 
A  writer  that  is  assigned  to  avionics,  for  example,  must  not  only  be  knowledgeable  about  each  of 
the  avionics  systems  available,  but  also  how  those  systems  interact  with  other  subsystems.  When 
writing  the  documentation  for  an  aircraft  that  has  system  A,  they  are  actually  writing  the 
documentation  for  the  hundreds  of  aircraft  with  that  same  system.  The  document  must  not  only 
reflect  the  data  applicable  to  that  system,  but  it  must  also  address  any  potential  interaction 
between  combinations  of  installed  subsystems. 
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2.1.4  Document  Lag. 


As  new  data  becomes  available,  the  writers  must  review  that  data  and  make  any  required 
additions  or  changes  to  the  manual.  If  multiple  changes  are  necessary,  they  are  worked  in  order 
of  importance  and  safety  relevance.  Source  data  continues  to  evolve  throughout  the  writing 
process,  but  at  some  point  in  time,  a  decision  must  be  made  to  finish  authoring  activity  for  the 
manual.  Additional  time  is  then  needed  to  compile  all  of  the  information,  format  and/or  print  it, 
and  send  it  to  the  operators.  Once  the  decision  point  is  reached,  the  document  is  said  to  be 
“locked”  and  no  changes  will  be  made.  The  “lockup”  date  is  usually  3  to  4  weeks  before  the 
intended  release  of  the  manual  or  revision.  Revision  schedules  are  either  driven  by  the  amount  of 
accumulated  content  or  a  calendar  cycle. 

At  the  point  of  lockup,  the  process  of  developing  the  next  revision  begins,  starting  with  any  data 
not  included  in  the  previous  revision.  Even  though  writing  has  ceased  for  that  release  of  the 
manual,  engineering  continues  to  progress  during  the  lockup  period.  The  airplane  manufacturer 
tries  to  synchronize  the  release  of  maintenance  data  with  the  delivery  of  a  new  airplane,  but  in 
some  cases,  design  engineering  data  or  supplier  data  is  released  after  the  lockup  period  for  the 
current  revision.  Consequently,  this  late  engineering  or  supplier  data  can  cause  a  “document  lag” 
in  which  the  released  maintenance  data  does  not  match  the  delivered  airplane  configuration. 
Maintenance  data  not  included  in  the  current  revision  of  the  manual  will  be  prioritized  for 
inclusion  into  a  later  revision.  If  safety-sensitive  data  or  maintenance  significant  data  (causing  an 
operator  economic  or  technical  difficulty)  emerges  between  the  release  of  the  current  and  next 
planned  revision,  manufacturers  have  established  a  process  to  provide  a  temporary  revision  until 
the  next  release  becomes  available. 

2.1.5  Distribution. 

Once  a  manual  or  manual  revision  is  completed,  it  must  be  converted  to  the  deliverable  format 
requested  by  the  operator  and  distributed.  For  domestic  operators,  the  distribution  is 
straightforward.  Postal  delivery  or  parcel  service  can  have  a  hard  copy  of  the  manual  to  a 
maintenance  facility  in  a  few  days  or  less.  From  that  point,  it  is  the  responsibility  of  the 
respective  maintenance  personnel  to  assure  the  new  information  is  added  to  the  manual  and 
disseminated  to  mechanics. 

For  international  operators,  the  distribution  process  can  be  slowed  weeks  or  even  months 
depending  on  the  efficiency  of  customs  procedures  it  needs  to  pass  through.  Non-English 
speaking  facilities  may  slow  down  the  process  further  by  translating  the  information  into  their 
native  tongue. 

2.1.6  Multiple  Manual  Formats. 

Although  the  size  of  the  published  manual  gives  a  good  indication  of  the  magnitude  of  the  task 
required  to  develop  one,  in  practice,  the  manuals  are  frequently  distributed  to  users  in  other 
media  formats.  Microfiche  or  microfilm  cassettes  are  common  media  options  for  distributing 
manuals  without  the  physical  space  requirements  of  paper.  Mechanics  can  view  film  versions  of 
the  manual  in  the  appropriate  viewer  and  print  the  pages  pertaining  to  the  job  they  are  assigned. 
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Although  they  are  smaller  and  easier  to  store,  film  versions  of  the  manual  are  identical  to  the 
paper  manual  and  differ  only  in  the  presentation  media.  Although  the  manual  can  be  distributed 
on  whatever  media  preferred  by  the  operator,  the  paper  copy  of  the  manual  remains  the  master 
reference. 

Digital  data  formats  are  slowly  replacing  other  media  formats.  Large  operators  prefer  the  ease 
and  flexibility  of  digital  manuals.  Large  amounts  of  electronic  data  can  be  easily  transferred  via 
network  or  distributed  on  CD-ROM.  Computerized  maintenance  data  can  be  easily  manipulated 
by  the  operator  and  integrated  into  company  manuals.  If  desired,  electronic  maintenance  data  can 
then  be  downloaded  to  a  notebook  or  hand-held  computer  for  use  by  a  field  mechanic  at  the 
aircraft.  Digital  data  does  not  suffer  from  delays  associated  with  printing  and  distribution  of 
paper  manuals  and  manufacturers  could  conceivably  distribute  daily  updates  via  the  Internet. 
Computerized  data  allows  for  the  inclusion  of  multimedia  enhancements  not  available  in  paper- 
based  manuals.  In  spite  of  its  potential  advantages,  the  implementation  of  computerized  data  is 
inconsistent  across  the  industry. 

To  avoid  potential  problems  arising  from  different  revision  schedules  for  paper  and  electronic 
data,  electronic  data  is  synchronized  to  the  slower  paper  publication  schedule,  eliminating  the 
potential  distribution  time  savings.  As  a  result,  the  full  benefits  of  the  electronic  medium  are 
currently  not  being  realized. 

Aside  from  the  problems  of  digital  versus  print  data,  supporting  multiple  data  mediums  requires 
additional  effort  to  verify  that  document  formatting  is  compatible  with  each  type  of  media 
supported  by  the  individual  manufacturer.  Each  aircraft  operator  may  request  the  manual  in  a 
different  media  to  match  existing  hardware  base  (e.g.,  microfiche  viewer,  tape,  CD-ROMs). 
Interactions  between  new  and  legacy  systems  (i.e.,  data  maintained  on  older  computer  databases) 
can  cause  unforeseen  irregularities  in  the  appearance  of  the  finished  document.  Finding  and 
correcting  these  problems  may  at  times  require  manually  reviewing  each  new  page. 

2.1.7  Document  Customization. 

Depending  upon  the  size  of  the  operator,  maintenance  manual  data  may  be  modified  for  the 
purpose  of  integrating  into  company  manuals.  In  some  cases,  these  operators  receive  the  manual 
data  in  an  electronic  format  (tape,  CD-ROM,  native  SGML  code)  and  then  modify  it  to  adhere 
with  company  procedures.  Alternatively,  the  operator  may  provide  details  to  the  manual 
developer  and  have  them  produce  the  manual  in  a  format  consistent  with  approved  company 
manuals.  Customization  in  this  case  may  include  company  procedures  for  handling  parts, 
completing  paperwork,  or  reporting  problems.  In  other  cases,  customization  may  extend  to  the 
way  procedures  are  carried  out.  When  an  operator  specifies  a  procedure  that  is  different  than  that 
outlined  by  the  manufacturer,  the  manufacturer  will  examine  it  to  determine  whether  it  is 
consistent  with  what  it  considers  to  be  safe  procedures.  If  the  operator  requests  procedures  to  be 
entered  that  have  not  been  approved  by  the  manufacturer,  those  procedures  may  be  included  in 
the  manual  with  a  notation  indicating  that  it  is  customer-originated  data. 
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Operator  customization  of  the  manual  can  create  problems  for  both  the  manufacturer  and  the  user 
of  that  manual.  From  the  mechanics  perspective,  manufacturer-generated  data  and  company¬ 
generated  data  are  often  indistinguishable.  Likewise,  it  is  sometimes  difficult  for  the 
manufacturer  to  have  a  clear  idea  of  the  quality  of  their  procedures  when  they  have  been 
modified  by  the  operator.  Another  example  of  this  mismatch  is  the  generation  of  the  work  task 
cards  or  job  aids  generated  from  the  manual  for  use  by  mechanics.  Many  manufacturers  provide 
job  aids  automatically  generated  directly  from  the  manual,  however  many  operators  develop  their 
own  version  of  job  aids.  User  experience  and  feedback  may  be  quite  different  depending  on  the 
source  of  this  data  and  it  may  not  be  immediately  apparent  which  party  is  responsible  for  an 
identified  problem. 

2.1.8  Feedback. 

Once  a  manual  or  revision  is  released,  further  changes  may  be  required  to  address  user  feedback. 
The  user  is  the  ultimate  check  of  the  quality  of  maintenance  information.  If  the  user  thinks  the 
manual  contains  erroneous  data,  unclear  directions,  or  difficult  procedures,  they  can  report  such 
problems  back  to  the  customer  service  department  of  the  manufacturer.  In  the  case  of  larger 
operators,  concerns  raised  by  mechanics  are  typically  channeled  through  the  operators 
engineering  department  before  they  are  passed  on  to  the  manufacturer. 

The  integration  of  user  feedback  is  primarily  a  reactive  process,  in  which  the  data  is  released,  and 
remains  unchanged  unless  problems  are  reported.  Except  in  the  case  of  safety  of  flight, 
responses  to  user  feedback  may  differ  based  on  the  unique  requirements  of  an  operator.  In  the 
case  of  customized  manuals,  if  one  operator  wishes  to  change  the  way  a  particular  procedure  is 
performed,  they  will  make  a  request  and  only  their  manual  will  be  changed.  The  requests  of  an 
operator  will  reflect  the  efforts  of  their  own  engineering  departments  and  are  typically  not  shared 
with  potential  competitors.  Manuals  for  a  new  aircraft  may  start  out  very  similar,  but  over  time 
the  unique  feedback  generated  by  the  mechanics  and  engineering  departments  of  each  operator 
cause  them  to  diverge.  Many  operators  invest  large  amounts  of  time  and  resources  into 
customizing  their  manuals  and  do  not  wish  that  information  to  be  shared  with  potential 
competitors,  so  this  process  is  repeated  with  each  operator,  requiring  the  manual  developer  to 
continually  address  similar  issues.  As  a  result,  manufacturers  are  not  only  providing  manuals  to 
support  each  aircraft  they  produce,  but  multiple  variations  of  that  manual  to  address  the  unique 
desires  of  each  individual  operator.  Another  downside  of  this  practice  is  that  potential 
improvements  in  the  manual  are  not  being  disseminated  throughout  the  industry.  The 
requirement  for  unique  user  feedback  is  primarily  true  of  operators  of  large  aircraft,  but  regional 
operators  may  make  similar  requests. 

In  an  attempt  to  prevent  the  repeated  occurrence  of  similar  errors  in  the  manual  development 
process,  some  manufacturers  have  developed  methods  for  receiving,  categorizing,  and  cataloging 
identified  problems.  Once  recorded,  problems  can  be  tracked  to  identify  trends  in  errors  that 
point  to  potential  shortcomings  in  a  manufacturer’s  manual  development  process.  Whether  or 
not  a  database  of  errors  is  maintained,  continual  monitoring  of  customer  feedback  is  a  critical 
part  of  providing  a  document  that  meets  user  expectations. 
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2.1.9  Aircraft  Customization. 


Once  an  aircraft  leaves  the  factory,  it  will  continue  to  be  modified  throughout  its  operating 
lifespan.  Operators  make  changes  and  updates  to  avionics,  interiors,  and  passenger  comfort 
equipment.  In  some  cases,  changes  can  be  more  drastic,  such  as  structural  modifications  or  cargo 
conversions.  There  is  considerable  variation  in  the  degree  to  which  maintenance  information  is 
updated  to  reflect  those  changes.  If  the  aircraft  was  returned  to  the  manufacturer  or  a  factory- 
authorized  maintenance  center  for  modification,  a  fully  integrated  manual  update  is  often 
included  in  the  total  cost  of  the  job.  For  in-house  changes,  or  work  done  by  third-party 
modification  centers,  very  little  maintenance  information  may  be  available.  If  the  operator 
wishes  to  have  these  changes  integrated  into  their  existing  manuals  they  can,  but  because  the 
rework  was  not  due  to  a  factory-initiated  change,  the  operator  must  shoulder  the  cost.  Many 
times  the  operator  elects  not  to  pay  to  have  the  information  incorporated  in  the  original  manual. 
In  such  a  case,  the  operator  may  be  given  a  generic  manual  supplement  pertaining  to  the 
modification.  This  supplement  may  contain  detailed  maintenance  information  or  be  limited  to  a 
basic  technical  description. 

The  manufacturer  may  be  impacted  by  the  continued  customization  in  two  ways.  First,  if  the 
operator  wishes  to  integrate  the  new  information  in  their  manual,  the  manufacturer  must 
communicate  with  the  modification  center  to  gather  all  the  necessary  information  pertaining  to 
the  operation  and  maintenance  of  the  new  system.  This  can  often  be  very  difficult,  depending  on 
the  quality  of  the  engineering  provided  by  the  modification  center.  Secondly,  if  the  operator 
chooses  not  to  involve  the  manufacturer,  the  manufacturer  may  still  be  impacted  because  they  are 
not  aware  of  the  actual  configuration  of  the  aircraft.  Consequently,  future  attempts  by  the 
manufacturer  to  improve  the  aircraft  or  manual  will  not  consider  the  potential  interaction 
between  factory-originated  changes  and  modifications  performed  by  a  third  party.  When  revising 
manuals,  the  author  will  write  procedures  based  on  the  known  aircraft  configuration,  usually  the 
configuration  at  time  of  delivery.  If  a  writer  changes  a  procedure  based  on  original  equipment 
manufacturer  (OEM)  procedural  or  engineering  changes,  it  may  be  inappropriate  for  the  unique 
configuration  of  the  customized  aircraft. 

2.1.10  Older  Aircraft. 

The  high  cost  of  providing  maintenance  manual  updates  and  revisions  has  led  most 
manufacturers  to  implement  some  form  of  revision  subscription  service.  In  order  to  continue 
receiving  updates  to  maintenance  manuals,  operators  are  usually  required  to  pay  for  a 
subscription  to  that  manual.  A  subscription  entitles  the  operator  to  receive  all  revisions  and 
updates  as  they  are  released.  Most  manufacturers  also  include  a  regular  newsletter  that  includes 
information  about  pending  revisions  and  model  information.  Operators  that  opt  not  to  pay  for  a 
subscription  receive  only  information  considered  to  directly  impact  the  safety  of  flight. 

By  far,  the  greatest  numbers  of  maintenance  manual  changes  come  early  in  the  life  of  a  new 
aircraft.  The  time  pressures  of  delivery  often  result  in  the  latest  engineering  changes  being  left 
out  of  early  versions  of  the  manual.  The  largest  numbers  of  problems  with  the  manual  are 
usually  identified  soon  after  its  release,  and  those  problems  are  addressed  in  early  revisions.  At 
some  point,  the  number  of  changes  begins  to  decline  and  level  off.  It  is  at  this  point  that  an 
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operator  may  feel  that  the  cost  of  maintaining  a  manual  subscription  is  not  warranted  by  the  small 
amount  of  change  in  each  revision  and  may  choose  to  suspend  the  manual  subscription.  Unless 
the  operator  chooses  to  later  pay  to  have  the  maintenance  manual  updated,  the  manual  will  reflect 
the  condition  of  the  aircraft  at  the  time  of  suspension  regardless  of  future  changes  to  parts 
information,  vendors,  or  recommended  procedures.  With  the  age  of  some  aircraft  still  in  service, 
it  is  not  unheard  of  to  find  operators  with  manual  subscriptions  that  have  been  suspended  for  20 
or  30  years.  In  most  cases,  these  aircraft  are  being  flown  by  foreign  airlines  or  have  been  leased 
to  smaller  operators.  Because  these  manuals  have  been  suspended,  it  is  unclear  whether  these 
operators  are  using  old  manuals  or  newer  documents  purchased  from  a  third-party  manual 
developer. 

From  the  perspective  of  the  manufacturer,  older  aircraft  present  a  particularly  costly  challenge. 
Once  an  aircraft  model  is  released,  the  maintenance  information  is  open  to  scrutiny  and  must 
continue  to  be  supported  until  the  last  aircraft  is  retired.  Unlike  most  departments  within  a 
manufacturer,  technical  writers  must  remain  knowledgeable  about  the  design  and  function  of 
systems  in  both  production  and  out-of-production  aircraft.  For  an  established  aircraft 
manufacturer,  this  may  require  them  to  support  manuals  for  aircraft  that  have  been  out  of 
production  for  decades. 

2.2  MAINTENANCE  TECHNICAL  DATA  ERRORS. 

When  asked  about  ways  of  improving  technical  manuals  most  writers/developers  emphasize  the 
development  of  procedures  to  eliminate  the  accidental  inclusion  of  incorrect  technical 
information.  Although  accidents  attributed  to  correctly  following  maintenance  procedures  that 
turned  out  to  be  erroneous  may  capture  public  interest,  other  more  frequent  and  potentially  more 
significant  sources  of  errors  receive  comparatively  little  attention. 

It  is  important  to  note  the  difference  between  the  academic  and  popular  definitions  of  error.  In 
academia,  error  is  understood  to  encompass  a  wide  variety  of  forms,  including  lack  of  clarity  and 
the  omission  of  information.  However,  when  technical  writers  are  questioned  about  errors,  their 
definition  of  an  error  is  typically  limited  to  the  inclusion  of  incorrect  technical  information  in  the 
manual.  Quality  control  procedures  tend  to  focus  on  identifying  incorrect  information, 
formatting  irregularities,  errors  in  grammar,  and  aircraft  configuration  errors.  Using  this  narrow 
definition  of  error  may  lead  one  to  overlook  other  important  factors  that  impact  the  user’s 
perception  of  the  usability  of  a  manual. 

Although  they  may  gamer  considerable  attention  when  they  occur,  the  cases  of  blatantly  incorrect 
information  being  incorporated  in  a  manual  are  relatively  small  when  one  considers  the  size  of 
the  document  and  the  amount  of  detailed  information  it  contains.  The  possibility  of  a  procedure 
being  written  that  is  unclear,  difficult  to  follow,  or  fails  to  represent  a  mechanics  mental  model  of 
the  task  is  more  likely  and  has  the  potential  to  be  of  equal  or  greater  consequence  than  incorrect 
information.  Poor  usability  of  documents  introduces  a  systemic  potential  for  error  due  to  the 
unpredictability  of  how  a  mechanic  interprets  the  manual  and  how  closely  this  interpretation 
adheres  to  the  intent  of  the  writer. 


12 


If  an  incorrect  procedure  is  outlined,  the  error  will  presumably  be  identified  and  corrected  in  the 
first  few  attempts  at  doing  that  task.  The  effect  of  a  poorly  written  procedure  however  may  be 
much  more  subtle;  for  example,  a  mechanic  may  think  they  are  performing  a  task  correctly  when, 
in  fact,  they  are  not  interpreting  the  procedure  in  the  way  intended  by  the  writer.  In  this  case,  an 
unclear  procedure  may  lead  to  similar  mistakes  being  repeated  each  time  the  offending  procedure 
is  attempted. 

In  cases  where  difficulties  with  a  procedure  may  be  more  salient,  the  user  may  be  more  apt  to 
abandon  the  documented  procedure  and  rely  on  their  own  judgment  to  perform  the  task.  Because 
the  procedure  may  be  technically  sound,  a  problem  resulting  from  confusion  with  or  failure  to 
use  the  manual  is  likely  to  be  attributed  to  incompetence  on  the  part  of  the  mechanic.  Ultimately, 
a  mechanic  may  be  reprimanded  or  fired  for  committing  actions  they  thought  were  correct  in 
light  of  documentation  that  was  difficult  to  follow  or  understand. 

When  attempting  to  assess  the  degree  of  error  present  in  maintenance  manuals,  it  is  important  to 
have  a  clear  understanding  of  what  constitutes  error.  Reason’s  model  of  active  and  latent  error 
[10  and  1 1]  has  been  given  considerable  mention  in  the  maintenance  error  research  literature  [7 
and  12].  The  term  active  error  is  used  to  describe  an  erroneous  action  or  violation  committed  by 
an  individual,  while  latent  error  refers  to  environmental  factors  that  may  contribute  to  error.  The 
oft-cited  “Swiss  Cheese”  model  of  error  illustrates  the  way  latent  errors  caused  by  working 
conditions,  management  policy,  and  organizational  communication  can  contribute  to  a  situation 
that  results  in  an  active  error  on  the  part  of  a  mechanic.  Manuals  can  indeed  be  a  source  of  latent 
maintenance  error,  but  this  model  is  not  particularly  well  suited  to  the  study  of  errors  in 
maintenance  manuals.  Maintenance  error  investigations  focus  on  the  failure  event  or 
inappropriate  action  of  the  maintenance  personnel.  Attempts  at  reducing  active  errors  require  an 
understanding  of  all  possible  errors,  so  that  either  the  individual  or  the  system  can  be  changed  to 
protect  against  those  possibilities.  Because  the  number  of  possible  errors  is  potentially  infinite, 
likely  errors  are  usually  determined  by  examining  previous  error  occurrences  [12].  Once  an  error 
is  identified,  investigators  must  work  backward  to  examine  all  of  the  potential  contributing 
effects  that  lead  to  its  occurrence.  They  must  determine  how  far  to  track  the  source  of  error  and 
estimate  a  degree  of  influence  for  each  contributing  factor  [13].  Errors  in  maintenance  manuals 
are  different  from  the  erroneous  action  of  an  individual  and  attempts  at  identifying,  cataloging, 
tracking,  and  reducing  errors  in  technical  information  need  to  take  a  different  approach. 

As  Rasmussen  points  out  [14],  the  goal  of  generating  a  taxonomy  for  classifying,  analyzing,  and 
addressing  existing  errors  in  a  system,  based  on  previous  failures,  is  best  suited  to  systems  that 
remain  reasonably  static  for  long  periods  of  time.  Tracking  the  reliability  of  a  system  requires  an 
accumulation  of  data  over  time,  and  if  a  change  is  introduced  into  the  system  without  being 
controlled,  the  accuracy  of  the  resulting  reliability  measure  will  be  suspect. 

The  procedures  used  to  develop  maintenance  documentation  have  been  relatively  stable  for  a 
long  time.  However,  these  procedures  have  recently  experienced  substantial  changes  resulting 
from  the  application  of  new  technology  and  changes  in  the  demographics  of  the  user  population. 
From  the  earliest  aircraft  until  the  advent  of  computers,  manuals  were  assembled  by  hand 
through  cutting  and  pasting  of  text  and  hand-drafted  illustrations.  Even  early  applications  of 
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computer  technology  to  the  development  process  operated  merely  as  a  more  efficient  version  of 
the  cut  and  paste,  paper  document  assembly.  Technology  is  changing  so  rapidly  that  in  many 
cases  legacy  information  is  not  migrated  to  new  systems  before  those  systems  are  replaced  by  the 
next  version.  Consequently,  the  way  manuals  are  developed  today  is  not  the  same  as  they  were 
last  year  or  will  be  a  year  from  now.  Traditional  attempts  at  cataloging  and  reducing  process 
errors  may  have  worked  well  with  the  transfer  of  blueprint  information  to  paper  documents,  or 
even  early  computer-based  authoring  systems,  but  are  inadequate  to  address  the  problems  caused 
by  rapidly  evolving  technology  and  a  changing  user  workforce. 

Because  the  process  of  developing  manuals  is  now  so  dynamic,  any  attempt  to  develop  a 
reliability  measure  must  take  into  account  both  the  changes  in  the  development  process  and  the 
target  audience.  Attempts  to  measure  reliability  using  static  measures  such  as  recording  the 
number  of  typographical  errors  or  incorrect  part  numbers  are  necessary  but  no  longer  sufficient. 
Simple  reliability  measures  may  provide  information  about  the  integrity  of  the  process  but  it  will 
not  indicate  how  well  the  manual  meets  the  needs  of  the  user  [14].  The  user  population  is  not 
static,  and  as  the  development  process  continues  to  evolve,  the  manual  writers  must  repeatedly 
verify  that  it  continues  to  meet  the  needs  of  the  user  population. 

The  information  contained  in  the  manual  must  be  correct  but  to  focus  on  technical  accuracy  alone 
is  to  inappropriately  simplify  the  issue.  Focusing  attention  on  matching  the  task  to  the  user 
would  go  beyond  process  error  to  include  the  cognitive  and  psychological  processes  guiding  the 
task  and  user.  Reason  outlines  a  process  for  developing  a  framework  of  error  that  includes 
“contextual  error  sources”  introduced  by  the  task  and  situation  and  “basic  error  tendencies”  of  the 
individual  [10].  As  Reason  points  out,  while  there  is  no  universally  applicable  classification  for 
error  that  will  meet  all  needs,  a  general  framework  of  error  can  be  defined  from  repeated  user 
testing  that  can  predict  the  majority  of  problems. 

The  technical  writers  responsible  for  maintenance  manuals  are  professionals  who  take  seriously 
the  potential  safety  impact  of  their  work.  It  is  fair  to  assume  that  their  intention  is  to  produce  the 
best  manuals  possible,  but  in  spite  of  these  intentions,  errors  occur.  This  is  important  to  note  at 
the  outset,  as  error  cannot  be  separated  from  intention.  The  notion  of  intention  is  based  on  two 
things:  (1)  an  expressed  goal  to  be  attained  and  (2)  a  clear  procedure  for  how  to  reach  that  goal. 
Volition  is  fundamental  to  the  definition  of  error,  and  therefore,  the  term  error  can  only  be 
applied  to  intentional  actions.  Intention,  in  this  case,  includes  intention  in  action  as  well  as  prior 
intention.  Errors  then  fall  into  one  of  two  broad  categories:  (1)  a  failure  of  actions  to  go  as 
intended  (Norman’s  slips  and  lapses  or  Reason’s  execution  failures)  or  (2)  a  failure  to  choose  the 
appropriate  actions  to  achieve  the  desired  outcome  (Norman’s  slips  or  Reason’s  mistakes 
failures)  [15  and  16].  In  either  case,  the  intended  outcome  is  not  reached. 

While  the  terms  used  in  the  error  research  literature  generally  refer  to  an  individual,  they  could 
effectively  be  applied  to  the  processes  of  industry.  In  the  process  of  developing  maintenance 
manuals,  there  can  be  failures  of  execution  (printing  mistakes,  formatting  errors,  etc.)  or  failures 
of  planning  (difficult  procedures,  overlooked  information,  etc.).  Failures  of  execution  are 
reduced  through  proofreading  techniques,  technical  monitoring,  and  software  document  checkers. 
Recently,  efforts  intended  to  reduce  process  error  in  manufacturing  have  been  applied  to  the 
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development  of  technical  documents.  For  example,  ISO  9000  certification  was  originally 
conceived  as  a  method  of  improving  the  consistency  and  quality  of  manufacturing  processes  but 
is  now  being  applied  to  the  development  of  data  as  well  [17].  The  reduction  of  failures  of 
execution  requires  tightly  controlled  procedures  for  data  transfer  and  error  checking.  If 
procedures  are  in  place  to  eliminate  error  in  the  transfer  of  data  from  its  point  of  origin  to  its 
entry  in  the  manual,  the  task  is  reduced  to  one  of  verifying  that  the  procedures  were  indeed 
followed.  If  execution  errors  do  occur,  it  then  becomes  a  matter  of  determining  whether  the 
procedures  were  adequate  or  whether  they  were  violated.  The  error  source  can  be  quickly 
identified  and  eliminated.  Execution  failures  are  the  most  visible  type  of  error  and  the  easiest  to 
control.  ISO  9000  certification  seeks  to  eliminate  execution  error  by  documenting  all  procedures 
and  then  enforcing  adherence  to  those  procedures  by  all  personnel.  In  its  simplest  form,  ISO 
9000  certification  can  be  reduced  to  documenting  what  is  done  and  doing  what  is  documented.  If 
problems  arise,  then  the  procedures  must  be  adjusted  to  correct  the  problem. 

Just  as  latent  error  is  the  more  difficult  type  of  maintenance  error  to  address,  planning  failures  are 
the  more  difficult  type  of  error  to  identify  and  eliminate  in  document  creation.  In  the  case  of  a 
planning  failure,  an  action  is  executed  as  intended,  but  when  the  action  is  followed  to  its 
outcome,  it  fails  to  produce  the  desired  result.  As  it  applies  to  maintenance  manuals,  a  planning 
failure  would  mean  that  a  procedure  is  technically  sound  but  is  misunderstood  by  the  mechanic 
or  fails  to  match  the  way  the  job  is  actually  performed.  Format  checkers  and  peer  reviews  may 
not  identify  planning  failures  because  the  information  may  be  accurate  and  appear  to  follow  a 
logical  sequence  but  cannot  be  performed  by  the  mechanic  (due  to  physical  constraints,  available 
tools,  etc.)  or  may  be  misinterpreted  by  the  mechanic. 

Planning  errors  can  be  further  separated  by  their  ultimate  outcome.  Although  the  development 
process  may  suffer  from  planning  and  execution  errors,  a  flawed  procedure  might  have  little 
negative  impact  due  to  some  level  of  forgiveness  or  error  tolerance  within  the  system.  In  the  case 
of  maintenance  manuals,  this  tolerance  is  afforded  by  the  experience,  skills,  and  knowledge  of 
the  mechanics.  Experience  allows  a  highly  skilled  mechanic  to  overcome  minor  problems  using 
their  expert  knowledge  to  identify  what  the  writer  intended  or  to  find  a  way  to  perform  the 
described  task.  If  the  user  lacks  this  expert  knowledge,  the  tolerance  for  error  is  decreased  and 
smaller  errors  have  a  greater  potential  for  negative  impact  on  the  system. 

In  short,  any  attempt  to  improve  the  way  in  which  manuals  are  developed  requires  a  thorough 
understanding  of  the  processes  used  by  the  manufacturers  and  the  problems  encountered  by  the 
users.  In  Phase  1,  the  focus  is  on  the  procedures  used  by  the  manufacturers  to  develop  aircraft 
maintenance  manuals. 

3.  EVALUATION. 

3.1  SURVEY  METHODS. 

This  phase  sought  to  gain  a  working  knowledge  of  the  issues  surrounding  the  development, 
revision,  and  distribution  of  aircraft  maintenance  technical  manuals  and  the  current  industry 
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procedures  that  address  those  issues.  This  information  was  gathered  through  the  cooperation  of 
multiple  aircraft  manufacturers  and  their  personnel,  including: 

•  Technical  writers 

•  Customer  service  representatives 

•  Engineers 

•  Illustrators 

•  Department  managers 

•  Manufacturer  representatives 

Information  was  colleeted  through  informal  interviews  and  directed  discussions.  The  topics 
covered  in  these  interviews  included: 

•  The  process  of  developing  and  revising  maintenance  manuals. 

•  The  systems  and  procedures  used  to  coordinate  information  from  numerous  sources 
within  the  organizational  structure  of  the  manufacturer. 

•  The  solieitation  and  inclusion  of  user  feedback  into  the  development  of  technical 
manuals. 

•  The  means  used  to  identify,  track,  and  reduce  error  in  fielded  technical  manuals. 

In  addition  to  these  direct  contacts,  participation  in  industry  conferences  provided  an  opportunity 
to  discuss  additional  industry  perspectives. 

Site  visits  were  conducted  at  each  of  the  participating  organizations.  These  visits  consisted  of  a 
series  of  meetings  over  the  course  of  several  days.  To  the  extent  possible,  site  visits  included 
demonstrations  of  procedures  and  technology  used  in  the  technical  publication  process.  Visits 
were  normally  arranged  through  a  technical  publications  manager.  In  addition  to  the  hosting 
manager,  interviews  included  employees  from  a  variety  of  areas  in  the  technical  manual  process. 
Researchers  were  provided  with  eopies  of  procedure  guidelines,  organizational  charts,  and 
related  documents  as  appropriate. 

The  industry  sample  included  five  aircraft  technical  manual  producers.  The  five  organizations 
included  manufacturers  and  modifiers  of  FAR  Part  25  aircraft,  representing  both  regional  and 
large  commereial  transport  models.  For  the  purposes  of  anonymity,  the  participating 
organizations  will  be  referred  to  throughout  this  report  as  companies  V,  W,  X,  Y,  and  Z. 
Because  of  the  limited  information  available  from  manufacturers  concerning  the  types  and 
number  of  identified  errors,  data  supplied  by  third-party  maintenance  facilities  was  used  to 
supplement  the  review  of  problems  identified  in  fielded  manuals. 
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3.2  INDUSTRY  ANALYSIS. 


3.2.1  Who  Writes  Manuals? 

The  manufacturers  differ  considerably  with  regard  to  what  they  consider  appropriate 
qualifications  for  a  technical  writer.  For  example,  company  X  hires  mostly  engineers  while 
companies  V,  W,  and  Z  hire  a  composite  of  certified  mechanics,  writers,  and  technicians,  while 
company  Y  hires  equal  numbers  of  engineers  and  former  mechanics.  The  make-up  of  the 
technical  writing  staff  is  potentially  important,  as  the  knowledge  base  of  the  writer  may  not 
match  that  of  the  user.  In  some  cases,  an  engineer’s  cognitive  representation  of  a  mechanical 
system  may  be  very  different  from  that  of  a  mechanic. 

3.2.2  New  Manual  Development. 

The  procedures  used  by  the  surveyed  companies  to  develop  manuals  differed  primarily  with 
regard  to  the  systems  used  to  communicate  between  engineers,  writers,  and  operators.  This 
section  is  summarized  in  table  1. 

TABLE  1.  NEW  MANUAL  DEVELOPMENT 


Company  V 

•  All  communication  and  transfer  of  engineering  information  is  handled 
through  a  centralized  computer  database. 

•  Engineering  and  technical  publications  are  separate  departments. 

Company  W 

•  Engineers  and  technical  writers  are  collocated  for  new  projects. 

•  Computers  are  used  for  authoring,  but  integrated  team  meetings  are  used  to 
facilitate  communication. 

Company  X 

•  Engineers  and  technical  writers  are  collocated  for  new  projects. 

•  Multiple  computer  systems  are  used  for  authoring. 

•  The  majority  of  information  transfer  is  handled  through  centralized  computer 
systems. 

•  Integrated  team  meetings  are  also  used  to  facilitate  communication. 

Company  Y 

•  Multiple  computer  systems  are  used  for  authoring. 

•  Information  transfer  is  handled  through  centralized  computer  systems  and 
face-to-face  meetings. 

Company  Z 

•  Computers  are  used  for  authoring. 

•  Memos,  face-to-face,  and  electronic  communications  are  used  to  transfer 
data. 

For  each  of  the  reviewed  manufacturers,  the  process  of  writing  a  new  manual  begins  during  the 
early  planning  stages  of  aircraft  development.  As  the  preliminary  details  of  the  aircraft  systems 
are  established,  descriptions  of  aircraft  systems  are  developed  for  use  in  the  manual.  To  the 
extent  possible,  a  framework  is  then  developed  with  the  anticipated  information  required  to 
maintain  those  systems.  This  framework  is  based  on  established  MSG  logic  and  any  similarity 
with  existing  models.  From  this  point  on,  any  differences  between  the  organizations  in  the  initial 
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development  of  manuals  reflect  their  unique  approaches  toward  communication  between  various 
groups  including  writers,  engineers,  and  customer  service. 

At  company  W ,  the  initial  development  of  the  aircraft  concept  involves  technical  manual  writers. 
When  a  new  project  is  first  proposed,  a  senior  writer  is  assigned  to  lead  manual  development  and 
is  involved  in  the  decisions  regarding  budget,  systems,  and  component  vendors.  The  senior 
writer  has  the  opportunity  to  set  requirements  for  the  data  that  vendors  and  suppliers  must 
provide.  The  senior  writer  is  also  part  of  an  integrated  product  team  that  tracks  deadlines, 
changes  to  engineering,  or  potential  problems.  The  product  group  is  a  means  for  immediately 
communicating  any  pertinent  information  to  all  affected  parties.  Communication  between 
engineers  and  writers  is  facilitated  by  first  assigning  a  writing  team  to  the  new  project  and  then 
having  them  physically  move  their  offices  to  collocate  with  the  development  engineers.  In  this 
way,  technical  writers  are  aware  of  proposed  engineering  changes  and  can  anticipate  how  those 
changes  may  impact  their  schedules.  A  more  subtle  effect  of  this  arrangement  is  that  technical 
writers  are  aware  of  the  issues  driving  engineering  changes  that  might  be  used  to  improve  the 
maintenance  procedures.  The  approach  used  by  company  W  is  in  contrast  to  the  computer-based 
solutions  adopted  by  other  manufacturers.  Computers  are  an  integral  part  of  the  writing  process 
at  company  W,  but  greater  importance  is  placed  on  maintaining  face-to-face  communication 
between  the  individual  involved  in  the  aircraft  development  process. 

Companies  V  and  X  employ  centralized  computer  databases  to  coordinate  the  writing  process. 
When  an  engineer  at  company  V  develops  new  data,  that  information  is  entered  into  the 
centralized  database  along  with  the  identity  of  the  responsible  engineer.  The  information  entered 
into  the  database  identifies  all  technical  drawings  that  were  modified  as  a  result  of  the  change. 
The  technical  writer  is  responsible  for  identifying  what  systems  may  need  to  be  modified  to 
reflect  the  change.  When  a  writer  begins  work  on  new  engineering  data,  their  name  is  also  added 
to  the  database.  Along  with  the  identities  of  the  writers,  information  is  entered  about  the  work 
hours  required  and  expected  completion  date.  One  of  the  primary  advantages  of  computerized 
database  tracking  is  the  ability  to  have  a  real-time  status  report  of  work  in  progress.  If  a  writer 
needs  information  about  the  way  in  which  an  engineering  change  will  interact  with  the  system 
they  are  responsible  for,  they  need  only  consult  the  database  and  call  the  engineer  responsible  for 
the  change  data.  If  there  is  a  potential  interaction  between  systems,  all  of  the  related  engineers 
are  easily  identified.  The  system  used  by  company  V  not  only  tracks  the  development  and 
distribution  of  data;  it  is  also  used  to  track  inventory.  A  side  benefit  of  this  use  is  that,  in  the 
event  that  a  writer  has  a  question  about  the  look  or  operation  of  a  part,  they  need  only  consult  the 
database  with  the  part  number  and  they  can  identify  the  actual  bin  location  of  the  part  and 
retrieve  a  sample  part  for  inspection. 

During  a  new  airplane  program,  company  X  also  has  its  technical  writers  collocate  with  design 
engineers  to  facilitate  accurate  and  rapid  maintenance  data  development.  The  operators  are  also 
asked  to  provide  feedback  concerning  the  usability  of  the  maintenance  data.  Although  they  use 
face-to-face  communication,  much  of  the  information  exchange  between  writers,  engineers,  and 
operators  is  handled  via  electronic  communication. 
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Company  X  uses  multiple  computerized  databases  based  on  a  variety  of  electronic  formats  to 
process  technical  data.  Engineering  data,  configuration  data,  illustrations,  change  requests,  and 
workload  assignments  are  all  transferred  via  centralized  computer  systems.  When  an  individual 
job  is  completed,  the  information  is  formatted  and  compiled  into  the  manual  automatically  by  the 
computer.  If  any  formatting  problems  are  encountered,  the  computer  reports  those  problems  and 
does  not  include  that  information  until  it  is  corrected.  The  software  used  by  company  X  is  a 
combination  of  in-house  and  contractor-developed  systems.  Multiple  versions  are  concurrently 
in  use  as  a  result  of  the  need  to  develop  software  to  support  new  aircraft  programs  while 
maintaining  existing  systems  to  continue  supporting  older  programs.  Because  of  the 
inefficiencies  resulting  from  multiple  incompatible  systems,  company  X  is  attempting  to  migrate 
to  a  single  new  system  thus  eliminating  the  problems  of  maintaining  and  integrating  information 
from  multiple  platforms. 

3.2.3  Configuration  Management. 

Configuration  management  within  the  surveyed  companies  differed  in  the  degree  of  manual 
customization  they  provide.  This  section  is  summarized  in  table  2. 

TABLE  2.  CONHGURATION  MANAGEMENT 


Company  V 

Manuals  include  all  maintenance  information  for  a  model  and  are  not  customized  to  a 
specific  aircraft. 

Company  W 

Manuals  include  all  maintenance  information  for  a  model  and  are  not  customized  to  a 
specific  aircraft. 

Company  X 

Manuals  are  customized  to  a  specific  operator. 

Company  Y 

Manuals  are  customized  to  a  specific  operator. 

Company  Z 

Manuals  include  all  maintenance  information  for  a  model  and  a  one-time  manual 
supplement  is  published  for  each  specific  aircraft. 

A  primary  reason  that  companies  may  be  confronted  with  maintaining  older  and  incompatible 
databases  is  due  to  the  need  to  support  aircraft  that  have  remained  in  service  longer  than 
anticipated.  These  databases  contain  the  configuration  information  for  each  airframe  produced 
by  the  manufacturer.  Before  any  change  is  made  to  a  manual,  the  configuration  information  must 
first  be  referenced  to  determine  how  that  change  may  impact  each  aircraft.  For  companies  V,  W, 
and  Z  that  offer  relatively  few  configuration  options  for  each  aircraft,  the  impact  of  configuration 
differences  on  manual  development  is  minor.  They  produce  a  single  manual  that  addresses  all 
aircraft  of  a  particular  model  type.  Equipment  differences  are  identified  by  aircraft  serialization, 
and  call-outs  are  used  in  the  manual  to  identify  these  differences.  Company  Z  also  provides  a 
manual  supplement  limited  to  cabin  interiors  and  passenger  amenities.  This  manual  supplement 
was  developed  primarily  as  a  customer  courtesy  and  is  not  updated  with  the  rest  of  the  manual. 
Errors  identified  in  the  configuration  management  often  stem  from  small  differences  between 
very  similar  systems  not  being  identified  through  serialization  changes. 
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In  order  to  satisfy  their  customers,  companies  X  and  Y  will  produce  an  aircraft  meeting  almost 
any  requested  configuration.  Therefore,  unlike  companies  V,  W,  and  Z,  they  invest  a 
considerable  effort  in  configuration  management  and  support.  This  requires  the  maintenance  and 
support  of  large  databases  and  associated  software.  Li  practice,  this  means  that  not  only  do  they 
produce  a  unique  manual  for  a  particular  model  but  also  for  each  operator  and  in  some  cases  each 
aircraft. 

3.2.4  Error  Tracking. 

The  surveyed  companies  differed  considerably  in  the  attempt  to  track  and  identify  the  source  of 
manual  errors.  This  section  is  summarized  in  table  3. 

TABLE  3.  ERROR  TRACKING 


Company  V 

• 

• 

Error  is  tracked  internally  but  not  traced  for  the  purpose  of  source  identification. 

A  database  is  maintained  for  errors  identified  by  external  sources. 

Company  W 

• 

An  error-tracking  program  is  being  developed. 

Company  X 

• 

• 

Internally  and  externally  identified  errors  are  recorded  and  tracked. 

Error  tracking  is  used  as  a  measure  of  production  performance. 

Company  Y 

• 

• 

Internally  and  externally  identified  errors  are  recorded  and  tracked. 

Error  tracking  is  used  as  a  measure  of  production  performance. 

Company  Z 

• 

• 

Error  tracking  is  limited  to  manual  revision  history. 

No  attempts  are  made  to  trace  error  causation. 

Because  of  the  amount  and  specificity  of  the  technical  information,  all  manufacturers  invest 
considerable  effort  in  verifying  the  accuracy  of  new  maintenance  procedures.  The  basic 
proofreading  process  is  similar  across  manufacturers;  however,  the  manufacturers  vary  markedly 
in  the  way  they  handle  errors  once  they  are  identified.  These  differences  stem  from  what  point  in 
the  development  a  manufacturer  chooses  to  track  errors.  The  first  point  for  tracking  errors  is  at 
the  time  of  inclusion  in  the  manual.  If  the  development  process  includes  a  formal  proofreading, 
errors  can  be  recorded  and  tracked  to  identify  trends.  If  there  appears  to  be  a  pattern  of  errors, 
new  procedures  or  training  can  be  developed  to  eliminate  this  trend.  Errors  identified  at  this 
stage  are  limited  to  problems  in  the  execution  of  the  writing  process. 

The  second  point  in  the  process  at  which  errors  can  be  tracked  is  after  the  document  is  released  to 
the  user.  User  feedback  allows  the  tracking  of  not  only  execution  errors  such  as  incorrect  part 
references  but  also  planning  errors  such  as  confusing  language  or  difficult  procedures. 
Identifying  and  tracking  errors  in  a  fielded  manual  requires  formal  procedures  for  customer 
communication  and  feedback. 

All  of  the  surveyed  manufacturers  have  systems  for  proofreading  new  data  before  it  is  cleared  for 
publication.  For  written  procedures,  the  first  level  of  proofreading  usually  consists  of  a  peer 
review.  Depending  on  the  manufacturer,  the  next  level  of  proofreading  includes  either  a  review 
by  a  group  lead  or  a  document  checker.  Illustrations  are  presented  to  the  requesting  writer  to 
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verify  that  the  drawing  meets  the  intention  of  the  writer.  Illustrations  are  then  checked  for 
nontechnical  errors  such  as  line  weights  and  call-outs.  The  level  of  formality  with  which  these 
procedures  is  performed  may  differ  between  companies,  but  the  basic  tasks  are  the  same.  Where 
companies  differ  considerably  is  in  the  tracking  of  the  errors  identified  during  the  proofreading 
process. 

While  all  of  the  surveyed  companies  have  error-checking  procedures,  only  companies  V,  X,  and 
Y  attempt  to  catalog  the  errors.  At  the  time  of  survey,  company  W  was  attempting  to  establish  an 
error  database.  For  those  that  catalog  errors,  it  is  generally  limited  to  process  errors,  or  errors 
found  before  the  release  of  manual  data.  Even  for  those  companies  that  do  attempt  to  catalog 
errors,  there  is  little  systematic  source  tracking  of  the  errors.  Company  V  maintains  a  database  of 
customer  service  reports,  cataloged  by  aircraft  model,  but  does  not  attempt  to  identify  the  error 
source.  Only  company  X  maintains  a  detailed  database  of  the  errors  and  attempts  to  track  their 
source. 

For  each  of  the  companies,  there  is  a  reluctance  to  maintain  detailed  records  of  past  errors 
because  of  fear  of  how  that  information  could  be  used.  Employees  are  wary  of  errors  being 
traced  down  to  the  level  of  the  individual  for  fear  of  punitive  repercussions  affecting  pay  or 
promotion.  The  companies  are  also  wary  of  maintaining  a  detailed  error  history  because  of 
exposure  to  liability  or  regulatory  action. 

3.2.5  Customer  Feedback. 

All  of  the  surveyed  companies  have  systems  for  receiving  and  responding  to  customer  feedback. 
This  section  is  summarized  in  table  4. 

TABLE  4.  CUSTOMER  FEEDBACK 


Company  V 

•  A  customer  service  hotline  is  maintained. 

•  Customer  feedback  is  entered  into  a  database  and  tracked  by  comment,  aircraft 
model,  and  planned  response. 

•  A  website  is  available  for  customer  feedback. 

Company  W 

•  A  customer  service  hotline  is  maintained. 

•  Users  are  solicited  through  written  and  phone  surveys. 

•  A  website  is  available  for  customer  feedback. 

Company  X 

•  Customer  service  is  handled  through  a  network  of  field  representations,  telephone 
hotline,  and  teletype  communications. 

•  Customer  site  visits  have  been  used. 

•  A  website  is  available  to  customers  that  allows  for  customer  feedback,  access  to 
all  maintenance  documents,  and  aircraft  information. 

Company  Y 

•  Customer  service  is  handled  through  a  network  of  field  representations,  telephone 
hotline,  and  teletype  communications. 

Company  Z 

•  A  customer  service  hotline  is  maintained. 

•  A  website  is  available  for  customer  feedback. 
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The  handling  of  customer  feedback  in  most  companies  is  similar  to  new  engineering  data.  After 
a  customer  reports  a  problem,  their  complaint  is  checked  for  (1)  accuracy  (2)  if  the  complaint 
involves  factual  data  or  practicality,  or  (3)  if  it  involves  rephrasing  or  modifying  a  procedure. 
Customer  support  personnel  are  usually  trained  mechanics  that  are  well  acquainted  with  the 
aircraft  they  support,  so  they  are  often  responsible  for  evaluating  the  content  of  a  customers 
comment.  Next,  the  data  is  routed  to  either  engineering  or  technical  publications  for  comment. 
For  those  manufacturers  that  utilize  Integrated  Product  Teams,  the  information  is  brought  to  the 
team  for  review.  Once  it  is  determined  that  a  change  is  warranted,  the  task  is  assigned  a  revision 
schedule.  Manufacturers  have  procedures  for  triage  of  revision  data  based  on  the  manual  chapter 
affected,  the  extent  of  the  change,  the  perceived  importance,  and  budget  considerations. 
Regardless  of  the  manufacturer,  exceptions  are  made  for  information  directly  impacting  the 
safety  of  flight.  Safety  of  flight  data  is  handled  immediately  and  will  often  result  in  the  issuing  of 
a  temporary  revision. 

Each  of  the  surveyed  companies  maintains  customer  support  networks  that  gather  feedback  from 
users  and  answer  questions.  Customer  feedback  can  be  gathered  through  mail,  telephone 
hotlines,  fax.  Teletype,  and  e-mail.  In  addition  to  direct  contact  with  the  manufacturer,  factory 
representatives  and  service  centers  are  available.  Typically,  problems  reported  through  hotlines 
or  customer  support  pertain  to  a  specific  issue  associated  with  the  manual  (e.g.,  a  question  about 
a  specific  procedure)  and  reveals  little  about  the  users  perception  of  the  overall  quality  of  the 
technical  documentation.  Companies  have  attempted  to  use  surveys  to  solicit  more  general 
information  about  how  they  might  better  meet  the  needs  of  the  users.  Phone  solicitations  and 
mass  mailing  have  been  used  to  obtain  user  feedback.  The  quality  of  this  information  is  often 
suspect  because  the  feedback  is  usually  not  very  specific  and  is  frequently  limited  to  blanket 
statements  of  like  or  dislike.  In  an  attempt  to  improve  the  quality  of  feedback  gathered  through 
user  survey,  on-site  surveys  and  interviews  have  also  been  performed.  Whether  it  is  a  function  of 
the  unexpected  nature  of  the  unsolicited  contact,  the  lack  of  salient  examples  of  problems,  or  low 
motivation,  the  average  response  to  manufacturer-initiated  surveys  adds  little  to  the  information 
already  gathered  through  user-initiated  contacts. 

3.2.6  Validation. 

Procedure  validation  is  only  used  to  a  limited  degree  within  the  surveyed  companies.  This 
section  is  summarized  in  table  5. 

Rather  than  relying  solely  on  user  feedback  to  identify  problems  with  new  or  significantly 
changed  procedures,  the  aircraft  manufacturers  have  used  validation  techniques  to  evaluate  the 
quality  of  the  procedures.  Validation  involves  actual  users  attempting  to  complete  a  procedure 
and  reporting  any  difficulties  encountered.  The  validation  is  performed  under  conditions  that 
replicate  the  working  environment  as  closely  as  possible  so  that  performance  can  be  expected  to 
be  similar  to  the  real  world.  Validation  is  an  excellent  means  of  testing  the  maintenance  data 
without  the  potential  for  costly  mistakes.  Procedure  testing  has  the  added  benefit  of  encouraging 
communication  between  the  user  and  writer.  A  procedure  may  appear  to  be  accurate  and  sound, 
but  when  a  user  attempts  it,  they  find  an  easier  way,  or  suggest  information  that  might  aid  a 
mechanic  in  completing  the  procedure.  In  some  cases,  safety  or  economic  limitations  preclude 
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the  validation  of  a  procedure  through  user  performance.  For  example,  many  troubleshooting 
procedures  cannot  be  validated  unless  a  specific  component  is  damaged.  To  simulate  a  damaged 
component,  or  to  actually  damage  a  component,  may  not  be  economically  feasible  or  may  impose 
a  potential  safety  risk.  In  these  cases,  validation  procedures  can  be  extended  to  include  simulated 
task  performance  and  user  analysis  of  instructions.  Simulation  would  involve  a  user  working 
through  a  task  in  accordance  with  the  maintenance  instructions  without  actually  performing  the 
task.  The  analysis  involves  a  user  reading  through  the  maintenance  instructions  with  the  purpose 
of  identifying  potential  problems  or  concerns.  Analysis,  simulation,  and  performance  can  be 
thought  of  as  a  set  of  validation  “tools”  to  be  applied  as  appropriate  in  light  of  the  constraints 
imposed  by  the  task  to  be  evaluated. 

TABLES.  VALIDATION 


Company  V 

•  Writers,  with  the  aid  of  service  center  mechanics,  can  verify  written 
procedures  or  descriptions. 

Company  W 

•  Writers  have  access  to  aircraft  during  production  if  they  want  to  verify  a 
written  procedure  or  description. 

Company  X 

•  All  scheduled  maintenance  procedures  pertaining  to  two  of  the  newest 
models  are  validated. 

Company  Y 

•  Procedures  are  evaluated  through  analysis,  simulation,  and  performance. 
Validation  is  applied  to  procedures  identified  as  problematic. 

Company  Z 

•  An  attempt  has  been  made  to  validate  new  procedures  but  is  often  based  on 
prototype  aircraft. 

All  of  the  companies  surveyed  have  used  some  form  of  procedure  validation,  but  most  use  it  in 
an  unsystematic  fashion.  Companies  V,  W,  and  Y  will  validate  procedures  that  a  writer  is  unsure 
of  or  that  have  gotten  negative  comments  from  field  users.  In  most  cases,  validation  testing  at 
these  companies  is  done  by  the  actual  writer  or  the  writer  and  a  company  service  center 
mechanic.  Company  X  has  been  validating  all  scheduled  maintenance  procedures  for  two  of  its 
latest  airplane  models.  The  choice  to  validate  maintenance  procedures  was  part  of  a  customer 
support  strategy,  aimed  at  improving  the  quality  of  new  aircraft  models.  Company  X  has  not 
retroactively  applied  the  extensive  validation  process  to  existing  airplane  programs.  Company  Z 
has  initiated  an  effort  to  validate  new  procedures,  but  has  found  that  development  schedules  have 
forced  testing  to  take  place  early  in  the  prototyping  stage,  resulting  in  much  of  the  information 
becoming  irrelevant  as  the  engineering  changes.  In  reality,  access  to  an  actual  aircraft  is  limited 
and  procedures  are  usually  written  using  engineering  documents  alone. 

3.2.7  Measures  of  Document  Quality. 

Each  of  the  surveyed  companies  lacked  an  objective  method  for  measuring  the  quality  of  manual 
data.  This  section  is  summarized  in  table  6. 

Each  of  the  surveyed  manufacturers  reports  customer  satisfaction  as  the  primary  measure  of 
document  quality.  Measures  of  customer  satisfaction  are  based  on  feedback  gathered  through 
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customer  support.  In  some  cases,  user  surveys  conducted  by  industry  journals  are  used  as  a 
benchmark  for  comparison  with  other  manufacturers.  Within  the  individual  companies,  quality 
is  also  judged  on  more  subjective  criteria  including  the  degree  to  which  the  writing  adheres  to  an 
established  look,  feel,  or  style.  Decisions  regarding  the  details  of  wording  and  writing  style  are 
left  to  the  judgment  of  the  writer,  and  through  the  feedback  of  lead  writers,  an  appreciation  for 
these  subjective  eriteria  is  conveyed  to  less  experienced  writers.  Whether  it  is  performed  by  a 
peer  or  a  formal  evaluation  procedure,  all  the  manufacturers  audit  revised  data  for  technical 
accuracy  in  terms  of  procedural  logic,  efficiency,  source  data  accuracy,  and  completeness. 
However,  it  should  be  emphasized  that  there  is  currently  no  independent  measure  of  manual 
quality  derived  using  psychometric  principles.  Instead,  quality  standards  derive  from  company 
history  and  what  users  have  accepted  in  the  past. 

TABLE  6.  MEASURES  OF  DOCUMENT  QUALITY 


Company  V 

•  New  procedures  are  reviewed  by  a  peer,  approved  by  a  lead  writer,  and 
proofread  for  grammar  and  typographic  accuracy. 

•  Vocabulary  is  limited  to  include  only  words  found  in  the  United  States  Air 
Force  dictionary. 

•  The  writing  process  is  ISO  9000  certified. 

Company  W 

•  New  procedures  are  reviewed  by  a  peer,  approved  by  a  lead  writer,  and 
proofread  for  grammar  and  typographic  accuracy. 

•  Attempts  are  being  made  to  become  ISO  9000  compliant. 

Company  X 

•  New  procedures  are  reviewed  by  a  peer,  approved  by  a  lead  writer,  and 
proofread  for  grammar  and  typographie  accuracy. 

•  Procedures  are  checked  for  adherence  to  the  simplified  English  standard. 

•  The  writing  process  is  ISO  9000  compliant  and  certification  is  expected. 

Company  Y 

•  New  procedures  are  reviewed  by  a  peer,  approved  by  a  lead  writer,  and 
proofread  for  grammar  and  typographic  accuracy. 

•  Procedures  are  checked  for  adherence  to  the  simplified  English  standard. 

•  The  writing  process  is  ISO  9000  certified. 

Company  Z 

•  New  procedures  are  reviewed  by  a  peer,  approved  by  a  lead  writer,  and 
proofread  for  grammar  and  typographic  accuracy. 

3.3  SIGNIFICANT  ISSUES. 

Although  no  recommendations  will  be  made  until  the  completion  of  this  project,  the  following 
issues  were  identified  during  Phase  1 . 

3.3.1  Inconsistent  Guidelines  for  Control  of  the  Development  Process. 

In  many  cases,  the  development  of  manuals  is  driven  by  “tribal  knowledge”  and  company  history 
rather  than  clearly  defined  standards.  As  a  result,  it  is  hard  to  implement  and  enforce  improved 
procedures.  At  the  time  that  this  report  was  written,  only  companies  V,  X,  and  Y  have  sought 
ISO  9000  certification  of  their  technical  writing  process.  ISO  9000  is  only  one  of  the  options  for 
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verifying  compliance  to  company  procedures  and  certainly  not  without  its  own  costs.  However, 
some  effort  is  required  to  verify  that  all  published  technical  information  adheres  to  the  same 
standard  and  is  generated  in  a  manner  consistent  with  established  company  procedures. 
Measures  of  quality  should  be  based  on  adherence  to  established  procedures  rather  than  a  lack  of 
customer  complaints.  Customer  response  should  drive  the  guidelines  used  to  develop  the 
manual,  rather  than  serve  as  measure  of  document  quality.  If  detailed  guidelines  are  maintained 
and  enforced,  the  source  of  any  resulting  error  could  be  easily  tracked  and  corrected. 

3.3.2  Use  of  Feedback. 

The  use  of  user  feedback  is  generally  limited  to  a  reactive  response  to  problems  as  they  are 
identified.  Users  function  as  the  final  stage  of  proofreading  for  the  manual,  with  the  downside 
that  they  can  only  perform  that  function  once  the  offending  procedure  has  been  attempted  and  a 
difficulty  is  encountered.  The  primary  shortcomings  of  using  user  feedback  are  that  (1)  users  are 
allowed  to  encounter  a  procedure  that  has  the  potential  to  result  in  an  unchecked  maintenance 
error,  (2)  errors  in  procedures  that  are  not  frequently  used  may  go  unidentified  for  a  long  period 
of  time,  and  (3)  the  user  must  identify  the  error  and  take  the  time  to  report  it.  This  could  be 
eliminated  by  testing  or  validating  the  quality  of  procedures  before  their  release. 

Except  for  isolated  cases,  the  validation  of  maintenance  procedures  is  not  a  part  of  the  standard 
development  process.  When  validation  is  performed,  testing  usually  involves  company 
employees  rather  than  representative  users.  The  testing  of  procedures  is  intended  to  identify 
problems  before  the  manual  is  distributed  to  users.  A  critical  part  of  usability  testing  is  the 
evaluation  of  procedures  under  real-world  working  conditions,  using  mechanics  that  are 
representative  of  the  user  population.  Company-employed  mechanics  fail  to  represent  field  users 
to  the  extent  that  they  are  part  of  company  culture.  Likewise,  if  a  consistent  pool  of  field 
mechanics  is  maintained  to  act  as  participants  in  validation  studies,  they  will  cease  to  represent 
the  broader  user  population  as  they  become  more  acquainted  with  the  culture  of  the 
manufacturer. 

When  designing  any  product,  the  ultimate  success  of  that  design  is  dependent  on  how  valuable 
the  intended  user  finds  it.  In  the  technology  sector,  usability  testing  has  become  a  erucial  part  of 
new  product  development.  At  the  heart  of  designing  a  usable  product  is  the  adage,  “Know  thy 
user.”  Whether  the  product  is  a  portable  electronic  device,  a  software  product,  or  a  maintenance 
manual,  it  should  be  designed  with  the  user’s  experience,  preferences,  and  abilities  in  mind.  If 
designers  fail  to  gather  this  type  of  information  about  their  users,  the  only  choice  they  are  left 
with  is  to  design  with  their  own  experiences  and  preferences  in  mind.  In  almost  every  case,  this 
results  in  a  drastically  misdirected  effort.  Technical  writers  spend  their  days  submersed  in 
engineering  drawings  and  the  writing  process.  Aircraft  mechanics  spend  their  days  submersed  in 
the  task  of  maintaining  aircraft;  to  them  a  maintenance  manual  is  merely  an  addition  to  the 
collection  of  tools  necessary  to  complete  a  task.  They  are  not  privy  to,  nor  do  they  have  the  time 
to  research,  all  of  the  information  available  to  writers.  What  is  dismissed  as  elementary  to  the 
writer  may  pose  a  serious  obstacle  to  the  mechanic. 
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3,3.3  Tracking  Manual  Quality. 


One  of  the  primary  goals  of  this  research  phase  was  to  develop  an  estimate  of  the  degree  of  error 
present  in  fielded  technical  manuals.  To  date,  this  has  not  been  accomplished,  primarily  because 
such  data  is  not  tracked  by  manufacturers  to  a  level  sufficient  for  making  such  an  estimate.  The 
majority  of  error  tracking  done  by  manufacturers  is  limited  to  those  errors  identified  before 
release.  Monthly  reports  of  illustration,  grammar,  and  configuration  errors  are  used  to  measure 
the  work  output  of  various  departments.  This  type  of  tracking  is  useful  for  maintaining  internal 
control  of  procedures  but  provides  no  information  about  the  quality  of  fielded  manuals.  As  they 
relate  to  fielded  manuals,  errors  identified  before  release  are  a  measure  of  success  rather  than 
failure. 

Preliminary  interviews  of  mechanics  at  project  partner  airlines  indicate  that  fielded  manuals 
contain  very  little  erroneous  data.  In  the  case  of  one  large  and  active  maintenance  facility, 
mechanics  report  finding  an  average  of  only  two-three  errors  per  year  in  the  entire  collection  of 
manuals.  Analyses  of  manufacturer  customer  feedback  and  error  databases  maintained  by  third- 
party  maintenance  facilities  reveal  similar  levels  of  technical  accuracy.  While  there  were  cases 
of  incorrect  part  numbers  and  torque  settings,  the  majority  of  technical  errors  appeared  to  have 
resulted  from  attempts  to  speedup  the  writing  process  using  computers.  For  example, 
eonfiguration  errors,  resulting  from  copying  text  from  the  manual  of  a  similar  aircraft  or 
formatting  errors,  were  caused  by  failing  to  update  image  links. 

Mechanics  that  were  asked  about  the  technical  accuracy  of  maintenance  procedures,  reported  that 
they  were  generally  very  good.  However,  when  asked  about  problems  encountered  while  using 
manuals,  mechanics  reported  having  encountered  awkward  procedures  and  confusing  diagrams. 
In  some  cases,  suggestions  were  made  to  add  additional  labels  warning  against  potential  mishaps; 
all  of  which  relate  to  larger  usability  issues,  rather  than  simply  verifying  the  accuracy  of  included 
data.  As  mentioned  previously.  Phase  2  of  this  project  will  explore  these  issues  in  greater  depth 
through  user  interviews.  In  addition  to  the  tracking  of  internally  identified  errors,  writers  should 
seek  to  track  usability  problems  encountered  by  users.  Responding  to  trends  in  user  problems 
will  lead  to  constant  improvement  in  the  manuals,  resulting  in  the  best  possible  product  rather 
than  one  that  is  merely  acceptable. 

3.4  FINAL  COMMENTS. 

Of  the  identified  issues,  none  are  meant  to  be  direct  criticisms  of  the  personnel  responsible  for 
developing  technical  manuals.  In  nearly  all  cases,  these  issues  were  offered,  in  one  form  or 
another,  by  the  technical  writers  themselves  as  ways  of  improving  the  technical  manual  process. 
To  simply  suggest  that  such  changes  be  added  to  the  existing  task  of  manual  development  would 
serve  to  overload  a  system  that  is  already  operating  at  capacity.  The  time  and  budget  constraints 
that  manufacturers  place  on  their  technical  writing  departments  has  forced  them  to  do  the  best 
they  can  with  what  they  have.  Writers  are  aware  of  the  potential  benefits  of  detailed  error 
tracking  and  procedure  validation;  they  simply  lack  the  resources  needed  to  enact  these  programs. 
An  underlying  concern  is  the  degree  of  influence  that  maintenance  technical  writing  departments 
are  given  within  their  respective  companies.  It  has  been  suggested,  that  during  times  of  financial 
difficulties,  technical  writing  departments  are  often  the  first  to  suffer  reduced  funding.  Although 
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it  is  outside  of  the  scope  of  the  current  study,  it  should  be  noted  that  many  of  the  surveyed 
companies  cited  difficulties  in  hiring  and  retaining  sufficient  numbers  of  qualified  writers. 
Typically,  technical  writers  do  not  command  salaries  as  high  as  other  areas  of  design,  and  as 
result,  it  is  harder  to  find  qualified  applicants  to  fill  writing  positions.  On  occasion,  writers  are 
even  lured  to  other,  higher  paying  jobs  within  their  own  company.  As  a  result,  time  and  effort  is 
required  to  train  and  supervise  writers  that  could  otherwise  be  dedicated  to  improving  manuals. 

Finally,  although  the  application  of  computers  has  transformed  the  manual  writing  process,  it  has 
created  an  unforeseen  problem  for  technical  writers  in  the  expectation  that  technical  documents 
can  now  be  generated  much  more  quickly.  Word  processing  software  has  made  some  areas  of 
manual  development,  such  as  spell  checking  and  the  handling  of  illustrations,  much  faster  and 
easier,  but  the  time  required  to  generate  manual  content  has  changed  very  little.  Technical 
writers  must  now  battle  the  perception  that  they  should  be  able  to  instantly  update  manuals  as 
new  engineering  is  released.  This  perception  further  aggravates  the  problems  created  by  the 
limited  budget  and  scheduling  resources  allotted  to  technical  publications  departments. 
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